Methods and systems for displaying virtual objects from an augmented reality environment on a multimedia device

ABSTRACT

Methods and systems are disclosed for displaying an augmented reality virtual object on a multimedia device. One method comprises detecting, in an augmented reality environment displayed using a first device, a virtual object; detecting, within the augmented reality environment, a second device, the second device comprising a physical multimedia device; and generating, at the second device, a display comprising a representation of the virtual object.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This patent application is a continuation of and claims the benefit ofpriority to U.S. patent application Ser. No. 16/847,998, filed Apr. 14,2020, the entirety of which is incorporated herein by reference.

TECHNICAL FIELD

Various embodiments of the present disclosure relate generally toaugmented reality user interfaces. More particularly the presentdisclosure relates to presenting a virtual object from an augmentedreality environment, onto a television set or other multimedia device.The present disclosure also relates to enabling electronic transactionsat the various interfaces and devices.

BACKGROUND

Augmented reality is an area of growing interest and development.Augmented reality (AR) is the process of adding computer-suppliedcontent, including images, video, text, and other data as layers overdisplayed images of the real world. For example, when a portable devicecaptures an image or video of a scene including objects, one or moreapplications installed on the device can add information about theobjects or other related contents over the displayed scene to enhancethe user's experience. The user may utilize the overlaid information andcontents to make informed decisions and/or for personal entertainment.At the same time, portable devices are increasingly able to cast theirdisplays onto other multimedia devices, e.g., television sets, laptops,vehicles, projectors, etc.

A desire thus exists to bring augmented reality virtual content to moredisplays. The background description provided herein is for the purposeof generally presenting the context of the disclosure. Unless otherwiseindicated herein, the materials described in this section are not priorart to the claims in this application and are not admitted to be priorart, or suggestions of the prior art, by inclusion in this section.

SUMMARY OF THE DISCLOSURE

One embodiment provides a computer-implemented method for displaying anaugmented reality virtual object on a multimedia device, comprising:detecting, in an augmented reality environment displayed using a firstdevice, a virtual object; detecting, within the augmented realityenvironment, a second device, the second device comprising a physicalmultimedia device; and generating, at the second device, a displaycomprising a representation of the virtual object.

One embodiment provides a system for displaying an augmented realityvirtual object on a multimedia device. The system may comprise one ormore processors; and a data storage comprising instructions which, whenexecuted by the one or more processors, cause the one or more processorsto perform a method comprising: detecting, in an augmented realityenvironment displayed using a first device, a virtual object; detecting,within the augmented reality environment, a second device, the seconddevice comprising a physical multimedia device; and generating, at thesecond device, a display comprising a representation of the virtualobject.

One embodiment provides a non-transitory computer readable medium fordisplaying an augmented reality virtual object on a multimedia device.The non-transitory computer readable medium may store instructionswhich, when executed by one or more processors, cause the one or moreprocessors to perform a method comprising: detecting, in an augmentedreality environment displayed using a first device, a virtual object;detecting, within the augmented reality environment, a second device,the second device comprising a physical multimedia device; andgenerating, at the second device, a display comprising a representationof the virtual object.

Additional objects and advantages of the disclosed embodiments will beset forth in part in the description that follows, and in part will beapparent from the description, or may be learned by practice of thedisclosed embodiments. The objects and advantages of the disclosedembodiments will be realized and attained by means of the elements andcombinations particularly pointed out in the appended claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the disclosed embodiments, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate various exemplary embodiments andtogether with the description, serve to explain the principles of thedisclosed embodiments.

FIG. 1 depicts a block diagram of an augmented reality (AR)-enabledelectronic transaction system, according to one aspect of the presentdisclosure.

FIG. 2A depicts an exemplary embodiment of a user device, according toone aspect of the present disclosure.

FIG. 2B depicts an exemplary embodiment of a content retrieval platform,according to one aspect of the present disclosure.

FIG. 2C depicts an exemplary embodiment of a user library, according toone aspect of the present disclosure.

FIGS. 3A-3C illustrates an exemplary context in which the techniquespresented in the present disclosure may be used.

FIGS. 4A-4G depict exemplary graphical user interfaces (GUIs) presentedby a user device for AR-enabled transaction and user collection ofvirtual objects.

FIG. 5A is a flowchart of an exemplary method of a user device promptinga display at a multimedia device, according to one aspect of the presentdisclosure.

FIG. 5B is a flowchart of an exemplary method of a multimedia devicegenerating a display based on a prompt from a user device, according toone aspect of the present disclosure.

FIG. 6 is a flowchart of an exemplary method of retrieving or generatingcontent for displaying a virtual object at a multimedia device,according to one aspect of the present disclosure.

FIG. 7 is a flowchart of an exemplary method of generating a transactionor payment interface involving a virtual object from an AR environment,according to one aspect of the present disclosure.

FIG. 8 is a flowchart of an exemplary method of processing a transactionassociated with a virtual object from an AR environment, according toone aspect of the present disclosure.

FIG. 9 illustrates an implementation of a computer system that mayexecute techniques presented herein.

DETAILED DESCRIPTION OF EMBODIMENTS

Various embodiments of the present disclosure relate generally toaugmented reality user interface and, more particularly, to enablingelectronic transactions and presenting virtual objects across disparateaugmented reality environments.

The subject matter of the present disclosure will now be described morefully with reference to the accompanying drawings that show, by way ofillustration, specific exemplary embodiments. An embodiment orimplementation described herein as “exemplary” is not to be construed aspreferred or advantageous, for example, over other embodiments orimplementations; rather, it is intended to reflect or indicate that theembodiment(s) is/are “example” embodiment(s). Subject matter may beembodied in a variety of different forms and, therefore, covered orclaimed subject matter is intended to be construed as not being limitedto any exemplary embodiments set forth herein; exemplary embodiments areprovided merely to be illustrative. Likewise, a reasonably broad scopefor claimed or covered subject matter is intended. Among other things,for example, subject matter may be embodied as methods, devices,components, or systems. Accordingly, embodiments may, for example, takethe form of hardware, software, firmware or any combination thereof(other than software per se). The following detailed description is,therefore, not intended to be taken in a limiting sense.

Throughout the specification and claims, terms may have nuanced meaningssuggested or implied in context beyond an explicitly stated meaning.Likewise, the phrase “in one embodiment” as used herein does notnecessarily refer to the same embodiment and the phrase “in anotherembodiment” as used herein does not necessarily refer to a differentembodiment. It is intended, for example, that claimed subject matterinclude combinations of exemplary embodiments in whole or in part.

The terminology used below may be interpreted in its broadest reasonablemanner, even though it is being used in conjunction with a detaileddescription of certain specific examples of the present disclosure.Indeed, certain terms may even be emphasized below; however, anyterminology intended to be interpreted in any restricted manner will beovertly and specifically defined as such in this Detailed Descriptionsection.

For simplicity, the description that follows will be provided byreference to a “payment vehicle,” “transaction vehicle,” or a “paymentcard,” which generally refers to any type of financial alternative tocash. As is to be clear to those skilled in the art, no aspect of thepresent disclosure is specifically limited to a specific type of paymentvehicle or payment card. Therefore, it is intended that the followingdescription encompasses the use of the present disclosure with manyother forms of financial alternatives to cash, including credit cards,debit cards, smart cards, chip-based payment cards, single-use cards,prepaid cards, electronic currency (such as might be provided through acellular telephone or personal digital assistant), and the like. Paymentvehicles or payment cards can be traditional plastic transaction cards,titanium-containing, or other metal-containing, transaction cards, clearand/or translucent transaction cards, foldable or otherwiseunconventionally-sized transaction cards, radio-frequency enabledtransaction cards, or other types of transaction cards, such as credit,charge, debit, prepaid or stored-value cards, electronic benefittransfer cards, a “virtual” card (e.g., in the form of a display on asmart phone), or any other like financial transaction instrument. In anyevent, the payment vehicles described herein communicate accountinformation (e.g., an account number or other account indicativeinformation) during a purchase event and/or payment or credittransaction.

Traditionally, merchants and other store owners may accept payment/userauthentication data (e.g., PIN, CVV, passcode, answer to a secretquestion, etc.) via point of sale (POS) devices that have buttonsarranged in a standardized format of alphanumeric characters. Thestandardized format may be seen in standard PIN Pad and keyboardlayouts. Authentication data that can be entered via PIN pads andkeyboards comprise a sequence of alphanumeric characters (and specialsymbols in some cases). Authentication data of this nature may be easilydetected by an observer, or may be stolen by a data thief hacking into acomputer system storing such data. Entry or provision of suchauthentication data may be replicated by simply entering the stolen datausing PIN pads, keyboards, etc.

Uses for augmented reality are rapidly being explored and developed.However, current augmented reality interfaces involve dedicatedaugmented reality services, platforms, mobile device apps, or devices.This means that augmented reality (AR) may not be readily accessiblewithout special AR devices or device capabilities. At best, AR apps maybe installed on user mobile devices. These mobile devices may thendisplay AR user interfaces.

The disclosed systems and methods provide an improvement to the currentviewing limitations of augmented reality experiences, by allowingvirtual objects shown in augmented reality experiences, to be shown onother multimedia devices. For example, a user may select a virtualobject from their augmented reality experience and opt to show thevirtual object on their television set. In this way, the virtual objectmay be viewed by other users. For example, a user may view an augmentedreality experience comprising an interior design of a room. An augmentedreality view may display, within the physical room that a user may bein, virtual representations of furniture or décor. A user may thenselect, for instance, a couch from the augmented reality view fordisplay on a physical television set in the room. The television set maythen display a representation of the selected couch, for the user (andother people in the room) to view and examine.

For instance, the user may zoom in and out on the representation of theselected couch, opt to view different color options of the couch, rotatethe couch, view different fabric/material options, etc. The display onthe television set may be updated to reflect the user's actions (e.g.,zooming, rotations, specification changes, etc.). The augmented realitydisplay may also include an updated display showing the virtual objectin accordance with the user's actions. The shared display between theaugmented reality environment and the physical multimedia device (e.g.,the television set) may be used to share an augmented reality experiencewith more than one user at a time. This sharing of the augmented realityexperience could be used to facilitate business meetings or forentertainment. These interfaces may be useful for closer examination ofvirtual objects, or for any application where access to multiple screensand displays may be helpful.

In one embodiment, the augmented reality experience may be provided byan AR app installed on or accessed by a user device. The user device mayinclude, for example, a personal mobile phone or wearable device. Thephysical multimedia device may include a television set, screen, ormonitor. The user device may link to the physical multimedia device viaany network, as discussed in greater detail at FIG. 1 . A display of avirtual object on the physical multimedia device may be prompted by auser selection of the virtual object within the AR experience. Thedisplay may include a representation of the virtual object, as well asobjects related to the virtual object, options related to the virtualobject, or actions related to the virtual object. If a selected virtualobject is a couch, for example, a related object may include a coffeetable or end table. An option related to the couch may include a sizeconfiguration, material, or any other design details. An action relatedto the couch may include purchasing of the couch. For this instance, adisplay may include transaction specifications for purchase of thecouch, e.g., price, weight, delivery information, color, geometricdimensions, promotional code(s), user review(s), warranty information,size information, user information, etc. Other actions may includedisplay options, e.g., rotation, zooming in and out, etc. Additionalactions may include modifications to a virtual object, for instance,truncating, cropping, replicating portions, inserting features (e.g.,joints or sections), etc.

Referring now to the appended drawings, FIG. 1 depicts a block diagramof an AR-enabled electronic transaction system 100, according to oneaspect of the present disclosure. In general, FIG. 1 depicts a paymentenvironment 110, a user device 130, a content retrieval platform 135, amultimedia device 140, and a merchant system 150, all connected vianetwork 160. Network 160 may include the Internet, but may also includeother networks such as a corporate WAN, cellular network, satellitenetwork, or combination thereof, for example. The network 160 may beemployed to enable data communications between the various entitiesillustrated in FIG. 1 .

In the AR-enabled electronic transaction system 100 of FIG. 1 , thepayment environment 110 may include acquirer processor server(s) 115 andfinancial institution(s) 125, connected via a payment network 120.Acquirer processor server(s) 115 may handle electronic transactionsbetween the financial institution of the consumer and that of a merchant(e.g., at merchant system 150). For example, an acquirer processorserver 115 may receive payment vehicle information and/or correspondingauthentication data, which was collected by the merchant system 150. Theacquirer processor server 115 may then request, by way of paymentnetwork 120, an electronic transfer of funds from a financialinstitution 125 associated with the payment vehicle and the user. Theacquirer processor server 115 may also transfer the funds received fromthe user-related financial institution 125, to a financial institution125 associated with the merchant system 150.

The user device 130 may include, but may not be limited to, an ARconsumer product (e.g., an AR headset, an AR eyewear, etc.), asmartphone, a mobile phone, a tablet computer, a laptop, a portablecomputer, a desktop computer, a smart watch, a wearable device, etc. Theuser device 130 may comprise a computing device consistent with orsimilar to the computing system depicted in FIG. 9 . The user device 130may capture and display an environment including one or more objects,and may receive additional prompts from the content retrieval platform135 to present objects over a displayed environment. The user device 130may also be used to display an augmented reality environment. Forexample as shown in FIG. 3A, an augmented reality environment 300 may bedisplayed on user device 130. The augmented reality environment 300 maycomprise a display shown to a user 305. The augmented realityenvironment 300 may include a virtual object, e.g., object 310, as wellas a physical object 315. Physical object 315 may include any physical,tangible object in the physical, real environment of the user 305. Forexample, the physical object 315 may include a multimedia device 140.The physical object 315 may also be any other objects sharing a spacewith the user 305, e.g., a building, a couch, a table, a vehicle, etc.

The content retrieval platform 135 may detect or identify objects in theaugmented reality environment displayed on user device 130. To detectobjects in the augmented reality environment, content retrieval platform135 may detect the user device 130 connected to network 160. Contentretrieval platform 135 may also detect or be in communication with anaugmented reality platform (not shown) providing the augmented realitycontent displayed on the user device 130. An augmented reality app(e.g., AR application 234 in FIG. 2A) may also provide augmented realitycontent displayed on the user device 130, and be in communication withcontent retrieval platform 135.

Alternately or in addition, content retrieval platform 135 may detect orreceive content packages associated with the augmented reality contentbeing played on the user device 130. In some cases, augmented realityplatforms, services, and/or apps may provide content packages comprisinginformation on objects related to the augmented reality displays. Forexample, for a given display, content retrieval platform 135 may receivea platform-provided package of virtual objects, each object tagged witha unique object identifier.

Alternately or in addition, content retrieval platform 135 may detectvarious virtual objects that were displayed by an augmented realityplatform, e.g., via image recognition. Content retrieval platform 135may then identify object identifiers associated with the recognizedvirtual objects. In some cases, virtual representations of the virtualobjects may be retrieved using the identified object identifiers. Forexample, content retrieval platform 135 may access databases or storageentities to retrieve representations of the virtual objects, based ontheir object identifiers. The representations may include a 3D model, 2Dimage, schematic drawing, text description, icon, or any depiction ofthe virtual object. In some cases, the databases or storage entities maybe associated with or provided by merchant system 150. In otherscenarios, the databases or storage entities may be any repositorystoring visualizations of one or more virtual objects. Thevisualizations may be directly tagged with the object identifiers, orwith any codes linked to the object identifiers.

The content retrieval platform 135 may further detect various contentthat could augment the scene or environment displayed by the user device130. For example, content retrieval platform 135 may provide paymentinterfaces or features for purchasing the virtual objects or itemsrelated to the virtual objects. FIGS. 3A and 3B include exemplarypayment features, including a prompt 317 for a payment transaction and aselection of payment vehicles 319 for a user to complete a transaction.

An electronic payment interface may be provided in various formats,including augmented reality environments. In one embodiment, anelectronic payment interface may be a web page where a user can providepayment vehicle information as well as the corresponding authenticationdata (e.g., PIN, password, CVV, answer to a secret question, biometricdata, etc.). In another embodiment, an electronic payment interface maybe a web-based application through which a user can provide a paymentvehicle (or payment vehicle information) as well as the correspondingauthentication data. In another embodiment, an electronic paymentinterface may entail a mobile application, e.g., installed on a userdevice 130. The mobile application may provide access to an electronicpayment interface for a user to enter payment vehicle information and/orauthentication data. Yet another electronic payment interface mayinclude a display of the multimedia device 140. For example, contentretrieval platform 135 may prompt at least a portion of an electronicpayment interface to be shown on a TV screen of a television multimediadevice 140. The electronic payment interface may include augmentedreality or virtual reality interfaces, elements, or features. Thecontent retrieval platform 135 may comprise a computing systemconsistent with, or similar to, that depicted in FIG. 9 .

In one embodiment, the content retrieval platform 135 may maintain astored collection of objects. For each object, content retrievalplatform 135 may have saved, at least one object identifier, and anyspecifications related to the object, e.g., weight, price, deliveryinformation, promotional code(s), user review(s), image(s) of theobject, size information, use information, material composition,nutritional information, dosage, serving size, ingredient information,warnings, warranty information, related products/objects, or acombination thereof. The content retrieval platform 135 may manage thestored objects via content manager module 242 and store the objectspecifications and/or virtual representations of the virtual objects viaAR content storage 244. Both the content manager module 242 and ARcontent storage 244 are described in more detail at FIG. 2B.

The content retrieval platform 135 may also store one or more userlibraries comprising virtual objects and/or icons corresponding to thevirtual objects. Each user library may be associated with a respectiveuser and comprise one or more virtual objects that have been displayedto the user. An instance of a user library 248 is described in moredetail at FIG. 2C.

The multimedia device 140 may include any home entertainment device,including a television set, speaker, projector, stereo, gaming device,etc. As shown in FIG. 3B, a user may select a given virtual object 310from an augmented reality environment 300 by “grabbing” or “dragging”the virtual object 310 from its original position in the augmentedreality environment 300. In some cases, the user may indicate theirselection by “dragging” the virtual object 310 onto the showing of thephysical object 315 (e.g., multimedia device 140) in the augmentedreality environment 300. This overlay of the virtual object 310 on thephysical object 315 may indicate the user's desire to display thevirtual object 310 at the physical object 315. As shown by FIG. 3C, adisplay of the virtual object 310 may then be provided at the physical,multimedia device 140 (outside of the augmented reality environment300). In other words, at FIG. 3C, a multimedia device 140 (for instancea user's television set) may show a representation of the virtual object310 from the augmented reality environment 300. In this way, even usersoutside of the augmented reality environment 300 can view virtual object310. User 305 may perform this operation to share parts of theiraugmented reality experience with other users in their physical space,or to better view and interact with the virtual object 310.

A user may also select a virtual object 310 for display by framing aphysical object 315/multimedia device 140 in their user device 130frame, and tapping or double-tapping on the virtual object 310. In suchan embodiment, the physical object 315 may not be part of the augmentedreality environment 300, for the virtual object 310 to be displayed onthe physical object 315. In another embodiment, user device 130 andmultimedia device 140 may be linked via network 160, e.g., by aBluetooth or WiFi connection. Once the connection is established, aselected virtual object 310 may be displayed on the multimedia device140 linked to the user device 130.

The merchant system 150 may include a remote point of service (POS)terminal 152 and/or a payment application 154. The POS terminal 152 andpayment application 154 may both be used to collect payment vehicleinformation, payment credentials, and/or payment authenticationinformation. The merchant system may send a transaction authorizationrequest by way of the network 160 to an acquirer processor server 115.The acquirer processor server 115 may then fulfill the transactionauthorization request by verifying the payment vehicle information,payment credentials, and/or payment authentication information, andprompting a transfer of funds for payment. The remote POS terminal 152of the merchant system 150 may be in communication with the user device130 over the network 160. In some embodiments, the merchant system 150may provide or prompt an electronic payment interface to be displayed atthe user device 130, e.g., using payment application 154.

It should be noted that, although the user device 130, content retrievalplatform 135, multimedia device 140, and merchant system 150, are shownas separate entities in FIG. 1 , all or a portion of these components(and the steps performed thereof) may be implemented in a singlecomputing device consistent with or similar to that depicted in FIG. 9 .The configurations specifically discussed herein regarding thearrangement of the components depicted in FIG. 1 are merely exemplary,and it would be obvious to a person of ordinary skill in the relevantart that different combinations of components may be implemented on asingle or multiple computing devices.

FIG. 2A depicts an exemplary embodiment of a user device 130 accordingto one aspect of the present disclosure. As alluded to above, the userdevice 130 may be a computing device consistent with or similar to thatdepicted in FIG. 9 . In addition, the user device 130 may comprisesensor(s) 232, an augmented reality (AR) application 234, a paymentvehicle application 236, and a user history module 238.

The sensor(s) 232 may include global positioning system (GPS) sensors,vision sensors (i.e., cameras), audio sensors (i.e., microphones), lightsensors, touch sensors, temperature sensors, radio frequency sensors,direction sensors (i.e., magnetic compasses, magnetometers, gyroscopes),and acceleration sensors (i.e., accelerometers). In one embodiment, thecontent retrieval platform 135 may detect objects using input receivedfrom sensors 232, e.g., audio or image-based input. The input mayindicate an item selection of an object or a selection to manipulate anobject. Exemplary manipulations may include rotating, zooming in andout, changing color (e.g., highlighting), changing dimensions,purchasing the object, searching for objects related to the selectedobject, etc. User input detected by sensors 232 may also cause contentretrieval platform 135 to initialize object detection or retrieval. Forexample, a user may motion with their user device 130 and the motion maybe captured via the direction and acceleration sensors 232 of the userdevice 130. Based on the motion, the content retrieval platform 135 maydetect object selection by a user and prompt display of the selectedobject at the multimedia device 140.

The AR Application 234 may be a single application or a group ofapplications configured to collect various contextual data using one ormore sensors 232 of the user device 130, process the collectedcontextual data, and provide an AR experience to the user based on theprocessed contextual data. For example, using a camera of the userdevice 130, the AR application 234 may receive a visual input thatrepresents the real physical environment, and the content retrievalplatform 135 may identify objects in the AR environment or receivedata/tags about objects in the AR environment.

The AR application 234 may receive contextual data from sensors of theuser device 130, such as, e.g., the GPS sensor, accelerometer, andgyroscope, in order to determine the location, distance, angle, and/ormotion of the identified objects relative to the camera of the userdevice 130, or the location, distance, angle, and/or motion of thecamera of the user device 130 relative to the identified objects. Datacomprising the determined location, distance, angle, and/or motion maybe referred to as location and orientation data in the presentdisclosure. The AR application 234 may then use the location andorientation data to adjust the size, orientation, and/or placement ofvirtual contents over the real environment displayed in the user device130. The AR application 234 may then overlay or superimpose the virtualcontents over the displayed environment based on the adjustedparameters.

In one instance, the content retrieval platform 135 may detect andretrieve virtual objects from content conveyed via the AR application234 of the user device 130, and prompt a display of the virtual objecton the multimedia device 140. User interactions with the virtual objectsmay be received as input via the sensors 232, e.g., zooming, minimizing,dragging, or rotating the virtual objects. The content retrievalplatform 135 and the AR application 234 may then work in conjunction toupdate the display, in response to the user input. For example, thecontent retrieval platform 135 may retrieve, find, or generateadditional views of the objects, according to the user input, and the ARapplication 234 may display the views.

The AR application 234 and content retrieval platform 135 may furthercast or project the AR application to generate a display on themultimedia device 140, where the display may be controlled via inputs atthe user device 130. For example, the user may zoom or rotate a virtualobject on a touchscreen of their user device 130, and the AR application234 may allow the zooming or rotation manipulation of the virtual objectto be shown at the user device 130 screen and/or at the multimediadevice 140. The displays may include full-screen displays, partialdisplays, overlays, pop-ups, taskbars, windows, or any combinationthereof.

The payment vehicle application 236 may provide security and encryption(e.g., tokenization, etc.) for personal information needed forelectronic transactions. For example, the payment vehicle application236 may comprise digital wallet systems such as GOOGLE PAY, APPLE PAY,SAMSUNG PAY, PAYPAL, VENMO, etc. In one embodiment and as shown in FIG.2A, the payment vehicle application 236 may be stored on the client side(i.e. user device 130) and may be fully compatible with electroniccommerce websites (i.e., merchant websites). In some embodiments, thepayment vehicle application 236 may be on the server side, and may beone that a merchant or an organization creates for/about a user andmaintains on its server(s). The personal information retained by thepayment vehicle application 236 may comprise a shipping address, abilling address, payment methods, account numbers, expiration dates,security numbers and answers, and any other information needed forauthenticating an electronic transaction.

The user history module 238 may include a stored information on objectsstored by the user, as well as the user's transaction or purchasehistory. For example, the user history module 238 may store a collectionof icons for each TV/radio-derived object selected and stored by a user.These icons may be accessed by a user at a later date, either inaugmented reality interfaces, additional payment/purchase scenarios, orjust for reference. The icons may further be shared via social media orfrom one user device to another user device, e.g., via text messages.The user history module 238 may include a user library 248, as describedbelow.

FIG. 2B depicts an exemplary embodiment of a content retrieval platform135, according to one aspect of the present disclosure. In oneembodiment, content retrieval platform may include a content managermodule 242 and an AR content storage 244. In one embodiment, contentmanager module 242 may manage tags and identifiers of any objects storedin AR content storage 244. AR content storage 244 may include acollection of predetermined objects, e.g., a listing of selectablevirtual objects shown via AR application 234.

The content manager module 242 may also perform or prompt objectdetection techniques to detect objects shown by AR application 234.Object detection techniques may include, for example, machine learningmethods, speech recognition, sound recognition, image recognition, alibrary of predetermined objects, a sponsored object listing, previoustransaction history associated with a user, or a combination thereof.Exemplary machine learning techniques may include natural languageprocessing (NLP), text classifiers, image recognition classifiers, deeplearning, convolutional neural networks, You Only Look Once (YOLO),regression problems, random forest, etc., using unsupervised machinelearning such as cluster algorithms, principal component analysis (PCA),etc., reinforcement learning, or any models for detecting objects invisual contents/files, such as documents, images, pictures, drawings,media files, etc.

As one example, content manager module 242 may use a convolutionalneural network (CNN) for object detection. For example, image data maybe input into the CNN and passed through various convolutional layersand pooling layers. In one embodiment, the content manager module 242may employ a CNN, which may apply a series of filters to the raw pixeldata of the image to extract and learn higher-level features, which themodel may then use for classification. The CNN may be composed of astack of convolutional modules that perform feature extraction, witheach module consisting of a convolutional layer followed by a poolinglayer. The last convolutional module may be followed by one or moredense layers that perform classification. The types of the CNN which canbe implemented with the embodiments contemplated herein may include aregion-based CNN (RCNN), fast RCNN, and faster RCNN. Any currently-knownor later-developed object detection techniques may be used to identifythe objects in a multimedia content. The content manager module 242 maystore any detected objects using the AR content storage 244.

The content manager module 242 may also retrieve or generate a virtualversion of a virtual object. For example, content manager module 242 mayretrieve an image file from the AR content storage 244, based on anobject identifier. The content manager module 242 may also generate orprompt an interface viewable from an augmented reality app installed ona user device 130.

The AR content storage 244 may store object contents 246, which maycomprise virtual objects corresponding to the objects shown by the ARapplication 234 of the user device 130, as well as detailed and/orrelated information pertaining to the virtual objects. The AR contentstorage 244 may also store one or more user libraries 248 comprisingvirtual objects and/or icons corresponding to the virtual objects, eachuser library being associated with a respective user and comprising oneor more virtual objects that belong to the user.

FIG. 2C depicts an exemplary embodiment of a user library 248 accordingto one aspect of the present disclosure. As shown in FIG. 2C, each userlibrary 248 may be a data structure storing a user identifier 250 inassociation with one or more virtual objects 252 a, 252 b that belong tothe user. Each user library 248 may thus be associated with a useridentifier 250 of the corresponding user. The content retrieval platform135 (e.g., the content manager module 242) may be configured to searchfor a library of virtual objects that belong to a particular user, usingthe user identifier of that user. Further, each virtual object (252 a or252 b) stored in the user library 248 may in turn be associated with atimestamp (254 a or 254 b), a duration (256 a or 256 b), and acorresponding icon (258 a or 258 b). The entire user library, or one ormore virtual objects and/or icons contained in the user library may thenbe returned to the AR application 234 of the user device 130. Thevirtual object(s) and corresponding icon(s) may then be stored inassociation with the user identifier in a local storage of the userdevice 130. The virtual object(s) and corresponding icon(s) stored atthe user device 130 may thus resemble the user library 248. In otherwords, the data stored in the local storage of the user device 130 maybe in sync with the user library 248.

FIGS. 3A-3C depict an exemplary usage of the content retrieval platform135 to render virtual objects in real-world settings. For example asshown by FIG. 3A, a user 305 may view a virtual object 310 in anaugmented reality environment 300 displayed on a user device 130. Theaugmented reality environment 300 may display the virtual object 310, aswell as a physical object 315 comprising a multimedia device 140. Theuser 305 may then opt to display an image of the virtual object 310 ontheir multimedia device 140 by selecting the virtual object 310. FIG. 3Billustrates an embodiment in which selection of virtual object 310comprises dragging the virtual object 310 until it overlays physicalobject 315. However, object selection may include any gesture, e.g.,swiping, clicking, double-clicking, touching, tapping, grabbing, orpinching the depiction of virtual object 310 on a user interface of userdevice 130. Object selection may also include voice or sound commands,motioning using the entire user device 130 (e.g., flipping, shaking, orrotating the user device 130), cues from a controller or other deviceconnected to the user device, etc. As shown by FIG. 3C, upon selectionof the virtual object 310, a representation of the virtual object 310may be shown on the physical object 315/multimedia device 140. Thedisplay at the multimedia device 140 may include any portion of the ARenvironment 300 shown on the user device 130, one or more selectedvirtual objects 310, as well as information related to any selectedvirtual objects 310.

The display at the multimedia device 140 may also include anymanipulations of the representation of the virtual object 310. Forexample, a user may perform manipulations of the virtual object 310 atthe user device 130, and the manipulations may be reflected in anupdated display at the multimedia device 140 (and/or the user device130). Exemplary manipulations may include rotating, zooming, changingcolor, or changing size of the virtual object 310. The updated displaymay then reflect the manipulation. For example, if a user rotates avirtual object 310 90° in the screen of their user device 130, therepresentation of the virtual object 310 at the multimedia device 140may show the virtual object 310 rotated 90° from its original position.

FIGS. 4A-4G depict exemplary graphical user interfaces (GUIs) presentedby the user device 130 for AR-enabled display and purchase of objectsfrom multimedia streams. The GUIs of FIGS. 4A-4G may be rendered by thecontent retrieval platform 135 identifying objects from the multimediadevice 140, and working in conjunction with the user device 130 andmerchant system 150.

FIG. 4A depicts an exemplary interface 400, in which a virtual object310 from an AR environment on a user device 130, may be displayed on amultimedia device 140. For example, object 310 may be displayed as animage on interface 400. Interface 400 is illustrated in the followingfigures as an interface on a user device 130, but interface 400 may alsocomprise a display on a physical multimedia device 140. In someembodiments, certain features of interface 400 may be displayed on userdevice 130 and other features of interface 400 may be displayed on aphysical multimedia device 140. For example, in one scenario, objectsrelated to virtual object 310 may be displayed on both user device 130and multimedia device 140, while selection options (described in furtherdetail below) or payment authentication commands may be displayed onlyon user device 130.

The interface 400 may include a selection option 405, e.g., for a userto purchase the object 310. The selection option 405 may include abutton or tab with a caption, e.g., “BUY.” Upon detecting selection ofuser selection of the button or tab, payment vehicle application 236compatible with the remote POS terminal 152 may be initiated.

As shown in FIG. 4B, the object 310 may further be displayed oninterface 400 with related content and corresponding selection options,e.g., a first purchase selection option 410 a with a first relatedobject 410 b, and a second purchase selection option 415 a with a secondrelated object 415 b. FIG. 4C shows an exemplary transaction interface410 c and FIG. 4D shows an exemplary transaction interface 415 c.Transaction interfaces 410 c and/or 415 c may overlay or accompany auser device interface 400. In one scenario, transaction interfaces 410 cand/or 415 c may be displayed after receipt of user input selecting apurchase selection option, e.g., selection option 410 a (from FIG. 4B).The transaction interfaces 410 c and/or 415 c may include user paymentcredentials and a confirmation request, e.g., an input button with acaption reading, “pay now” or “confirm payment.” Upon the user selectingthe input button, the AR application 234 may, by way of the paymentvehicle application 236, transmit the transaction information to themerchant system 150 (or the remote POS terminal 152 of the merchantsystem 150). The merchant system 150 may then generate a transactionauthorization request based on the received transaction information, andtransmit the transaction authorization request to an acquirer processorserver 115.

In some cases, transaction interfaces 410 c and/or 415 c may also beconfirmed or modified. For example, transaction interfaces 410 c and/or415 c may include an option or prompt for user 305 to confirm/modify thedisplayed transaction information (e.g., user information comprisinguser name, user ID, payment account number, etc., which may be retainedby the payment vehicle application 236 securely, transaction amount,etc.) and complete a payment in a more convenient and prompt manner.FIG. 4E includes a display which may include option 420 to purchase apackage of objects, e.g., virtual object 310, related object 410 b, andrelated object 415 b.

FIG. 4F presents a display having an exemplary user history menu 425 ofpreviously collected or purchased items of the user. Icons or images ofpreviously selected or purchased objects from a multimedia stream may bedisplayed in positions 430 a-430 e. The user history menu 425 maycomprise a side pane or side bar. Menu 425 may appear briefly to confirmselection or purchase by showing an image of the object 310 (along withicon(s) representative of any other virtual object(s) that have beenselected/purchased by the user 305). Menu 425 may also fold or scrollinto a tab or minimized icon, so as not to obstruct the entire screenview. The tab or minimized icon may permit a user to re-open, expand, ordisplay the menu 425, in response to user input. Exemplary user inputmay include touching or swiping the tab or minimized icon, which theuser 305 can touch and swipe to the other side to re-open/expand theside pane menu 425. The menu 425 may also permit a user to delete orclear icons from the display.

Purchase information or related objects may be derived from thepreviously collected or purchased items displayed in user history menu425. In some cases, content manager module 242 may receive a searchrequest and determine, for a user, whether a selected item for purchasehas previously been purchased, e.g., by using a user library 248. If so,the user history menu 425 may simply refresh or update a priorentry/display. If the object was not previously purchased, contentmanager module 242 may provide an icon or image to be used in futuredisplays of menu 425.

FIG. 4G presents an exemplary pairing of user interfaces between a userdevice and a separate screen display. In the example at FIG. 4G, adrag-and-release motion may prompt a display refresh or clear-screencommand at a user device, e.g., user device 130. The display refresh orclearing of a screen command at the user device 130 may prompt areciprocal clear-screen/clear-display action at a different device orscreen paired to the user device 130. For example, a display at amultimedia device 140 may return to a blank screen after a user makes adrag-and-release motion at their user device 130. FIG. 4G illustrates auser making a dragging motion across a user device screen (e.g., asshown in panels 1 and 2) and a release motion (shown at panel 3). Panel4 shows a blank screen, illustrating a refreshed or cleared display.This display may be devoid of any virtual objects, as shown at panel 4.A remote screen (e.g., panel 5) may also refresh to a blank display. Theremote screen may be comprised of a display at multimedia device 140.

While FIG. 4G describes a display refresh as returning a screen to ablank state, a refresh may also return a virtual object representationto its initial state, prior to any user manipulations, or “undo” one ormore manipulations. Motions or commands in addition to drag-and-releasemay initiate screen refreshes between a user device 130 and multimediadevice 140. For example, a user may select a “clear screen” or “delete”prompt (not shown), use a voice command, “shake” their user device 130,or use any variety of predetermined gestures, audio, motion, or promptsto reset a screen.

As previously discussed, the disclosed embodiments may allow for a morefluid use experience between personal user devices and a sharedmultimedia device. FIG. 5A is a flowchart of an exemplary method ofenabling a user device (or content retrieval platform) to depict avirtual object from an AR display, at multimedia device. FIG. 5B is adepicts an exemplary method of enabling a multimedia device (or contentretrieval platform) to display a virtual object received from an ARenvironment. FIG. 6 includes an exemplary method of creating a displayof a virtual object.

FIGS. 7 and 8 depict exemplary methods for completing transactionsassociated with displayed virtual objects. For example, FIG. 7 describesone method of generating a display of transaction specifications for auser to select. FIG. 8 describes a method of approving payment andcompleting a purchase transaction associated with a virtual object.

FIG. 5A is a flowchart of an exemplary method 500 of enabling electronictransactions and presentation of virtual objects across disparatedevices, according to one aspect of the present disclosure. Method 500may be performed by the content retrieval platform 135 or by a userdevice 130. In one embodiment, step 501 may include detecting, in anaugmented reality environment, a virtual object. The augmented realityenvironment may be displayed using a first device, e.g., a mobile phoneof a user. Step 503 may include detecting a multimedia device. In oneembodiment of step 503, the multimedia device may be within theaugmented reality environment displayed using the first device. Forexample, the augmented reality environment may include physical objectscaptured by imaging capabilities of the first device, as well as,virtual objects created as virtual features of the augmented realityenvironment. The multimedia device may comprise a physical, tangibledevice captured by the imaging capabilities of the first device. Aspreviously discussed, the multimedia device may comprise a televisionset, or any other consumer media device with a visual interface. Themultimedia device may comprise a second device, different from the firstdevice.

Alternately or in addition, step 503 may include detecting a network(e.g., network 160) connected to the first device, and identifying amultimedia device connected to the same network. Step 503 may furtherinclude establishing a connection, pairing, or link between the firstdevice and the multimedia device, e.g., via a Bluetooth™, WiFi™,Zigbee™, wireless, Near Field Communication (NFC), radio frequencycommunication, or any other connection possible between two devices.

Step 505 may include prompting a first display of the virtual object atthe detected multimedia device. For example, step 505 may includegenerating, at the second device, a display comprising a representationof the virtual object from the augmented reality environment. In somecases, step 505 may include detecting a user selection and prompting thedisplay of the virtual object on the multimedia device in response tothe user selection. The user selection may include a voice command, or agesture, e.g., a swiping, clicking, double-clicking, touching, grabbing,or pinching motion across a depiction of the virtual object in theaugmented reality environment. Display of the virtual object at themultimedia device may be generated in response to the user selection.Step 505 may also include displaying the augmented reality environmenton a mobile device (first device) separate from the multimedia device.In one embodiment, the virtual object may be displayed simultaneously ata mobile device and at the multimedia device, sequentially displayed atthe mobile device and at the multimedia device, or displayed only on themultimedia device once the display of the virtual object at themultimedia device is initiated.

Step 505 may also include prompting a display of at least a portion ofthe augmented reality environment of the multimedia device. For example,the augmented reality environment may initially be displayed on aninterface of a user device, e.g., a mobile device. Upon initiating adisplay of the virtual object of the user device augmented realityenvironment onto a multimedia device, step 505 may further includeprompting a display of at least a portion of the augmented realityenvironment onto the multimedia device as well. Such a depiction mayprovide a visual context of visual aid of the virtual object, on themultimedia device. In some cases, the portion of the augmented realityenvironment may include background scenic features of the environment,or user prompts. User prompts may include prompts for payment ortransaction authentication. FIG. 7 discusses payment prompts in moredetail.

Step 507 may include receiving user input comprising an interaction withthe virtual object. The user input may be received via a mobile device,e.g., the mobile device displaying the augmented reality environment.User input may also be received via a controller associated with themultimedia device. Exemplary user input may include a selection torotate, zoom in, zoom out, change color, change dimensions of thevirtual object, purchase the object, view items related to the object,display the object with one or more other objects, or a combinationthereof. Step 509 may include prompting an updated display of thevirtual object at the multimedia device. The updated display maycomprise a zoomed in, zoomed out, color variation, or geometricallymodified view of the virtual object.

FIG. 5B is a flowchart of an exemplary method 520 of displaying avirtual object from an augmented reality environment, according to oneaspect of the present disclosure. Method 520 may be performed by thecontent retrieval platform 135 or by a multimedia device 140. Step 521may include receiving, at a multimedia device, a prompt to display avirtual object from an augmented reality environment. The virtual objectand augmented reality environment may be displayed using a first device,e.g., a mobile device. The prompt may be received from a mobile deviceor a mobile device in connection with a content retrieval platform 135.For example, step 521 may include detecting an electronic network anddetecting a mobile device connected to the electronic network. Theprompt to display the virtual object may be conveyed via the electronicnetwork by the detected mobile device. In some embodiments, step 521 mayinclude receiving the prompt directly at the multimedia device. In otherembodiments, step 521 may include receiving the prompt and thendetecting a physical multimedia device.

Step 523 may include generating, at the multimedia device, a firstdisplay of the virtual object. Step 525 may include receiving one ormore commands including an updated view of the virtual object. Thecommands may be received from the detected mobile device. Alternately orin addition, the commands may be received from a controller, a mobiledevice, a voice command, a gesture, or a combination thereof. Tofacilitate receipt of the commands, some embodiments of step 525 mayinclude generating a view control menu including options for a user toswipe, click, double-click, touch, grab, or pinch the first display ofthe virtual object, and/or generating a prompt for the user to provide avoice command.

Step 527 may include generating a display of the updated view of thevirtual object. For example, the display of the updated view of step 527may be generated based on a user response to a generated view controlmenu. The display of the virtual object may be accompanied by a paymentinterface for purchase of the virtual object.

FIG. 6 is a flowchart of an exemplary method 600 of displaying a virtualobject from an augmented reality environment, according to one aspect ofthe present disclosure. Method 600 may be performed by the contentretrieval platform 135. Step 601 may include identifying one or morevirtual objects in an augmented reality environment. In someembodiments, the content retrieval platform 135 may include or operateas a virtual reality or augmented reality content platform, providingthe virtual environment accessible by a user. In such a case, step 601may include differentiating between virtual objects in a virtual oraugmented reality setting, versus background displays. For example,virtual objects in an augmented reality display of a room may include acouch, a lamp, or a table. The background display may include the wallsof the room. As another example, virtual objects in an augmented realityenvironment depicting a forest may include virtual objects, e.g.,camping gear, a lantern, or food supplies, and the background mayinclude a depiction or captured imagery of a campsite.

Step 603 may include determining, for each of the identified virtualobjects, an object identifier. The object identifier may be specific tothe object, e.g., the couch, lamp, table, camping gear, lantern, or fooditem. Step 605 may include retrieving, based on the object identifier, astored representation of the virtual object. The stored representationmay include a 3D model, a 2D image, a schematic diagram, a photographicimage, an icon, or any stored depiction of the virtual object. Thestored representation may be a representation stored in a databasemaintained by the content retrieval platform 135, or any data storageentity accessible by content retrieval platform 135. For example,merchant system 150 may maintain a database of representations ofobjects that may be shown in the augmented reality environment. Merchantsystem 150 may also tag each of the representations using the objectidentifier, so that the representations may be retrieved in step 605.Multiple merchant systems or data storage entities may contribute to thevirtual objects that may be displayed in an augmented realityenvironment. A single augmented reality environment may include ordisplay virtual objects from one or more merchant systems or datastorage entities. In some embodiments, the representations may beupdated by the merchant system 150 or data storage entities. In suchcases, step 605 may include determining multiple differentrepresentations for a single virtual item, or retrieving differentrepresentations of the same virtual object at different times.

Step 607 may include detecting a multimedia device and casting theretrieved representation of the virtual object to the detectedmultimedia device. For example, a 3D model of a couch may be retrievedfrom a furniture merchant system, based on the object identifier of avirtual version of the couch in an augmented reality environmentdisplayed on a user's mobile phone. The 3D model of the couch may thenbe cast to a user's television.

FIG. 7 is a flowchart of an exemplary method 700 of displaying atransaction interface involving the virtual object from an augmentedreality environment, according to one aspect of the present disclosure.Method 700 may be performed by the content retrieval platform 135, inconjunction with user device 130 and/or multimedia device 140. In oneembodiment, step 701 may include generating a search request for one ormore transaction specifications associated with the virtual object(e.g., from method 600). The search request may include an identifierassociated with the virtual object.

Step 703 may include receiving, from a server, transactionspecifications associated with the identifier from the search request.The server may include a second content platform or a serveroperating/accessing AR content storage 244. Exemplary transactionspecifications may include object specification, weight, price, deliveryinformation, promotional code(s), user review(s), image(s) of theobject, size information, use information, material composition,nutritional information, dosage, serving size, ingredient information,warnings, warranty information, or a combination thereof.

Step 705 may include generating a display comprising the receivedtransaction specifications. The display may include a depiction of thevirtual object. For example, the display may include a payment interfaceand step 705 may be used to perform, at the user device, multimediadevice, or augmented reality environments, payment authentication for apurchase of or related to the virtual object. Step 705 may also includestoring the depiction of the object as a virtual object file as part ofthe user's file history. The user may access this file history andvirtual object at a later time, either via a user device or multimediadevice. Exemplary steps of completing a payment transaction using thedisplay of transaction specifications is described in FIG. 8 .

FIG. 8 is a flowchart of an exemplary method of 800 of conducting apayment transaction for a selected virtual object from an augmentedreality environment or display, according to one aspect of the presentdisclosure. Step 801 may include receiving a user selection to initiatea transaction associated with an object from a multimedia stream. Step803 may include presenting, at a user device or at a multimedia device,a transaction interface. The user device may comprise a personal mobiledevice and the multimedia device may comprise a monitor or televisionset. The transaction interface may include purchase information relatedto the object, including, product specifications (e.g., size, quantity,dimensions, etc.), user reviews, price, shipping information and cost,related objects, etc. The transaction interface may also include menu ofoptions for payment, e.g., a list of payment vehicles related to a userdevice or multimedia device, including credit card, bank, or app-basedcredit accounts. The transaction interface may be overlaid orsuperimposed on a display of the user device or multimedia device (e.g.,as shown by transaction interface 410 c in FIG. 4C and transactioninterface 415 c in in FIG. 4D).

Step 805 may include receiving, at a user device (or via a multimediadevice), transaction information, e.g., payment credentials and/or userinput indicating a desire to purchase the object. Step 805 may alsoinclude retrieving previously-generated or stored user accountinformation, or requesting that a user enter payment information. Forexample, AR application 234 may work in conjunction with a paymentvehicle application 236 to present the transaction interface. Thetransaction interface may allow the user to confirm/modify the displayedtransaction information and make an electronic payment.

Step 807 may include transmitting, to an acquirer processor server(e.g., acquirer processor server(s) 115), a transaction authorizationrequest based on the received transaction information. The transactionauthorization request may be generated based on the transactioninformation received, e.g., from the user device 130. The transactionauthorization request may comprise information needed by downstreamentities (e.g., acquirer processor server 115, payment network 120,financial institution 125) to route and/or authorize the transaction. Atstep 809, the transaction information may be transmitted to a merchantsystem (i.e., merchant system 150 in FIG. 1 ). This step may beperformed to confirm inventory of the object or to check whether themerchant system has any particular payment credential/security checks,loyalty programs, promotions/price reductions, etc.

Step 811 may include receiving a transaction authorization response,e.g., from a merchant system 150 or an acquirer processor server 115. Atstep 813, the transaction authorization response may be transmitted tothe user device 130 to complete a transaction. For example, the userdevice 130 may receive the transaction authorization response andcomplete the purchase by prompting payment within the paymentenvironment 110. Completion of a transaction in this context may includeproviding payment for purchasing an object. The transactionauthorization response may authorize payment transfer from the user'saccounts (and financial institution(s) 125) to a merchant system (e.g.,merchant system 150).

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specification,discussions utilizing terms such as “processing,” “computing,”“calculating,” “determining”, analyzing” or the like, refer to theaction and/or processes of a computer or computing system, or similarelectronic computing device, that manipulate and/or transform datarepresented as physical, such as electronic, quantities into other datasimilarly represented as physical quantities.

In a similar manner, the term “processor” may refer to any device orportion of a device that processes electronic data, e.g., from registersand/or memory to transform that electronic data into other electronicdata that, e.g., may be stored in registers and/or memory. A “computer,”a “computing machine,” a “computing platform,” a “computing device,” ora “server” may include one or more processors.

FIG. 9 illustrates an implementation of a general computer systemdesignated 900. The computer system 900 can include a set ofinstructions that can be executed to cause the computer system 900 toperform any one or more of the methods or computer based functionsdisclosed herein. The computer system 900 may operate as a standalonedevice or may be connected, e.g., using a network, to other computersystems or peripheral devices.

In a networked deployment, the computer system 900 may operate in thecapacity of a server or as a client user computer in a server-clientuser network environment, or as a peer computer system in a peer-to-peer(or distributed) network environment. The computer system 900 can alsobe implemented as or incorporated into various devices, such as apersonal computer (PC), a tablet PC, a set-top box (STB), a personaldigital assistant (PDA), a mobile device, a palmtop computer, a laptopcomputer, a desktop computer, a communications device, a wirelesstelephone, a land-line telephone, a control system, a camera, a scanner,a facsimile machine, a printer, a pager, a personal trusted device, aweb appliance, a network router, switch or bridge, or any other machinecapable of executing a set of instructions (sequential or otherwise)that specify actions to be taken by that machine. In a particularimplementation, the computer system 900 can be implemented usingelectronic devices that provide voice, video, or data communication.Further, while a single computer system 900 is illustrated, the term“system” shall also be taken to include any collection of systems orsub-systems that individually or jointly execute a set, or multiplesets, of instructions to perform one or more computer functions.

As illustrated in FIG. 9 , the computer system 900 may include aprocessor 902, e.g., a central processing unit (CPU), a graphicsprocessing unit (GPU), or both. The processor 902 may be a component ina variety of systems. For example, the processor 902 may be part of astandard personal computer or a workstation. The processor 902 may beone or more general processors, digital signal processors, applicationspecific integrated circuits, field programmable gate arrays, servers,networks, digital circuits, analog circuits, combinations thereof, orother now known or later developed devices for analyzing and processingdata. The processor 902 may implement a software program, such as codegenerated manually (i.e., programmed).

The computer system 900 may include a memory 904 that can communicatevia a bus 908. The memory 904 may be a main memory, a static memory, ora dynamic memory. The memory 904 may include, but is not limited tocomputer readable storage media such as various types of volatile andnon-volatile storage media, including but not limited to random accessmemory, read-only memory, programmable read-only memory, electricallyprogrammable read-only memory, electrically erasable read-only memory,flash memory, magnetic tape or disk, optical media and the like. In oneimplementation, the memory 904 includes a cache or random-access memoryfor the processor 902. In alternative implementations, the memory 904 isseparate from the processor 902, such as a cache memory of a processor,the system memory, or other memory. The memory 904 may be an externalstorage device or database for storing data. Examples include a harddrive, compact disc (“CD”), digital video disc (“DVD”), memory card,memory stick, floppy disc, universal serial bus (“USB”) memory device,or any other device operative to store data. The memory 904 is operableto store instructions executable by the processor 902. The functions,acts or tasks illustrated in the figures or described herein may beperformed by the programmed processor 902 executing the instructionsstored in the memory 904. The functions, acts or tasks are independentof the particular type of instructions set, storage media, processor orprocessing strategy and may be performed by software, hardware,integrated circuits, firm-ware, micro-code and the like, operating aloneor in combination. Likewise, processing strategies may includemultiprocessing, multitasking, parallel processing and the like.

As shown, the computer system 900 may further include a display unit910, such as a liquid crystal display (LCD), an organic light emittingdiode (OLED), a flat panel display, a solid-state display, a cathode raytube (CRT), a projector, a printer or other now known or later developeddisplay device for outputting determined information. The display 910may act as an interface for the user to see the functioning of theprocessor 902, or specifically as an interface with the software storedin the memory 904 or in the drive unit 906.

Additionally or alternatively, the computer system 900 may include aninput device 912 configured to allow a user to interact with any of thecomponents of system 900. The input device 912 may be a number pad, akeyboard, or a cursor control device, such as a mouse, or a joystick,touch screen display, remote control, or any other device operative tointeract with the computer system 900.

The computer system 900 may also or alternatively include a disk oroptical drive unit 906. The disk drive unit 906 may include acomputer-readable medium 922 in which one or more sets of instructions924, e.g. software, can be embedded. Further, the instructions 924 mayembody one or more of the methods or logic as described herein. Theinstructions 924 may reside completely or partially within the memory904 and/or within the processor 902 during execution by the computersystem 800. The memory 904 and the processor 902 also may includecomputer-readable media as discussed above.

In some systems, a computer-readable medium 922 includes instructions924 or receives and executes instructions 924 responsive to a propagatedsignal so that a device connected to a network 160 can communicatevoice, video, audio, images, or any other data over the network 160.Further, the instructions 924 may be transmitted or received over thenetwork 160 via a communication port or interface 920, and/or using abus 908. The communication port or interface 920 may be a part of theprocessor 902 or may be a separate component. The communication port 920may be created in software or may be a physical connection in hardware.The communication port 920 may be configured to connect with a network160, external media, the display 910, or any other components in system900, or combinations thereof. The connection with the network 160 may bea physical connection, such as a wired Ethernet connection or may beestablished wirelessly as discussed below. Likewise, the additionalconnections with other components of the system 900 may be physicalconnections or may be established wirelessly. The network 160 mayalternatively be directly connected to the bus 908.

While the computer-readable medium 922 is shown to be a single medium,the term “computer-readable medium” may include a single medium ormultiple media, such as a centralized or distributed database, and/orassociated caches and servers that store one or more sets ofinstructions. The term “computer-readable medium” may also include anymedium that is capable of storing, encoding, or carrying a set ofinstructions for execution by a processor or that cause a computersystem to perform any one or more of the methods or operations disclosedherein. The computer-readable medium 922 may be non-transitory, and maybe tangible.

The computer-readable medium 922 can include a solid-state memory suchas a memory card or other package that houses one or more non-volatileread-only memories. The computer-readable medium 922 can be arandom-access memory or other volatile re-writable memory. Additionallyor alternatively, the computer-readable medium 922 can include amagneto-optical or optical medium, such as a disk or tapes or otherstorage device to capture carrier wave signals such as a signalcommunicated over a transmission medium. A digital file attachment to ane-mail or other self-contained information archive or set of archivesmay be considered a distribution medium that is a tangible storagemedium. Accordingly, the disclosure is considered to include any one ormore of a computer-readable medium or a distribution medium and otherequivalents and successor media, in which data or instructions may bestored.

In an alternative implementation, dedicated hardware implementations,such as application specific integrated circuits, programmable logicarrays and other hardware devices, can be constructed to implement oneor more of the methods described herein. Applications that may includethe apparatus and systems of various implementations can broadly includea variety of electronic and computer systems. One or moreimplementations described herein may implement functions using two ormore specific interconnected hardware modules or devices with relatedcontrol and data signals that can be communicated between and throughthe modules, or as portions of an application-specific integratedcircuit. Accordingly, the present system encompasses software, firmware,and hardware implementations.

The computer system 900 may be connected to one or more networks 160.The network 160 may define one or more networks including wired orwireless networks. The wireless network may be a cellular telephonenetwork, an 802.11, 802.16, 802.20, or WiMAX network. Further, suchnetworks may include a public network, such as the Internet, a privatenetwork, such as an intranet, or combinations thereof, and may utilize avariety of networking protocols now available or later developedincluding, but not limited to TCP/IP based networking protocols. Thenetwork 160 may include wide area networks (WAN), such as the Internet,local area networks (LAN), campus area networks, metropolitan areanetworks, a direct connection such as through a Universal Serial Bus(USB) port, or any other networks that may allow for data communication.The network 160 may be configured to couple one computing device toanother computing device to enable communication of data between thedevices. The network 160 may generally be enabled to employ any form ofmachine-readable media for communicating information from one device toanother. The network 160 may include communication methods by whichinformation may travel between computing devices. The network 160 may bedivided into sub-networks. The sub-networks may allow access to all ofthe other components connected thereto or the sub-networks may restrictaccess between the components. The network 160 may be regarded as apublic or private network connection and may include, for example, avirtual private network or an encryption or other security mechanismemployed over the public Internet, or the like.

In accordance with various implementations of the present disclosure,the methods described herein may be implemented by software programsexecutable by a computer system. Further, in an exemplary, non-limitedimplementation, implementations can include distributed processing,component/object distributed processing, and parallel processing.Alternatively, virtual computer system processing can be constructed toimplement one or more of the methods or functionality as describedherein.

Although the present specification describes components and functionsthat may be implemented in particular implementations with reference toparticular standards and protocols, the disclosure is not limited tosuch standards and protocols. For example, standards for Internet andother packet switched network transmission (e.g., TCP/IP, UDP/IP, HTML,HTTP) represent examples of the state of the art. Such standards areperiodically superseded by faster or more efficient equivalents havingessentially the same functions. Accordingly, replacement standards andprotocols having the same or similar functions as those disclosed hereinare considered equivalents thereof.

The embodiments herein allow for augmented reality interfaces to bedisplayed on physical consumer devices, e.g., television sets andmonitors. Such experiences increase fluidity between augmented realityenvironments and physical environments. For example, the various displayoptions allow for augmented reality experiences to be shared betweenusers. As another example, the ability display and manipulate objects ona multimedia device (and larger screen) may facilitate user interactionwith virtual objects from augmented reality environments. In addition,the disclosed embodiments increase the functionality and uses ofmultimedia devices that may already exist in a user's physicalenvironment. The present embodiments decrease the limitations ofaugmented reality in being bound to personal devices/environments andincrease possible user/group interactions with virtual objects.

It will be understood that the steps of methods discussed are performedin one embodiment by an appropriate processor (or processors) of aprocessing (i.e., computer) system executing instructions(computer-readable code) stored in storage. It will also be understoodthat the invention is not limited to any particular implementation orprogramming technique and that the invention may be implemented usingany appropriate techniques for implementing the functionality describedherein. The invention is not limited to any particular programminglanguage or operating system.

It should be appreciated that in the above description of exemplaryembodiments of the invention, various features of the invention aresometimes grouped together in a single embodiment, figure, ordescription thereof for the purpose of streamlining the disclosure andaiding in the understanding of one or more of the various inventiveaspects. This method of disclosure, however, is not to be interpreted asreflecting an intention that the claimed invention requires morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive aspects lie in less than allfeatures of a single foregoing disclosed embodiment. Thus, the claimsfollowing the Detailed Description are hereby expressly incorporatedinto this Detailed Description, with each claim standing on its own as aseparate embodiment of this invention.

Furthermore, while some embodiments described herein include some butnot other features included in other embodiments, combinations offeatures of different embodiments are meant to be within the scope ofthe invention, and form different embodiments, as would be understood bythose skilled in the art. For example, in the following claims, any ofthe claimed embodiments can be used in any combination.

Furthermore, some of the embodiments are described herein as a method orcombination of elements of a method that can be implemented by aprocessor of a computer system or by other means of carrying out thefunction. Thus, a processor with the necessary instructions for carryingout such a method or element of a method forms a means for carrying outthe method or element of a method. Furthermore, an element describedherein of an apparatus embodiment is an example of a means for carryingout the function performed by the element for the purpose of carryingout the invention.

In the description provided herein, numerous specific details are setforth. However, it is understood that embodiments of the invention maybe practiced without these specific details. In other instances,well-known methods, structures and techniques have not been shown indetail in order not to obscure an understanding of this description.

Similarly, it is to be noticed that the term coupled, when used in theclaims, should not be interpreted as being limited to direct connectionsonly. The terms “coupled” and “connected,” along with their derivatives,may be used. It should be understood that these terms are not intendedas synonyms for each other. Thus, the scope of the expression a device Acoupled to a device B should not be limited to devices or systemswherein an output of device A is directly connected to an input ofdevice B. It means that there exists a path between an output of A andan input of B which may be a path including other devices or means.“Coupled” may mean that two or more elements are either in directphysical or electrical contact, or that two or more elements are not indirect contact with each other but yet still co-operate or interact witheach other.

Thus, while there has been described what are believed to be thepreferred embodiments of the invention, those skilled in the art willrecognize that other and further modifications may be made theretowithout departing from the spirit of the invention, and it is intendedto claim all such changes and modifications as falling within the scopeof the invention. For example, any formulas given above are merelyrepresentative of procedures that may be used. Functionality may beadded or deleted from the block diagrams and operations may beinterchanged among functional blocks. Steps may be added or deleted tomethods described within the scope of the present invention.

The above disclosed subject matter is to be considered illustrative, andnot restrictive, and the appended claims are intended to cover all suchmodifications, enhancements, and other implementations, which fallwithin the true spirit and scope of the present disclosure. Thus, to themaximum extent allowed by law, the scope of the present disclosure is tobe determined by the broadest permissible interpretation of thefollowing claims and their equivalents, and shall not be restricted orlimited by the foregoing detailed description. While variousimplementations of the disclosure have been described, it will beapparent to those of ordinary skill in the art that many moreimplementations and implementations are possible within the scope of thedisclosure. Accordingly, the disclosure is not to be restricted exceptin light of the attached claims and their equivalents.

What is claimed is:
 1. A method for prompting an updated display of avirtual object at a multimedia device, the method comprising: detecting,by a user device, the virtual object in an augmented reality environmentdisplayed; detecting, by the user device, the multimedia device withinthe augmented reality environment; prompting, by the user device, adisplay of the virtual object at the multimedia device; receiving, bythe user device, a user input comprising an interaction with the virtualobject; and prompting, by the user device, the updated display of thevirtual object at the multimedia device based on the user input.
 2. Themethod of claim 1, wherein the user device comprises an augmentedreality device connected to the multimedia device via a virtual network.3. The method of claim 1, wherein the interaction is a user selection ofthe virtual object.
 4. The method of claim 3, wherein the user selectioncomprises a touch gesture or a voice command.
 5. The method of claim 1,wherein the user input is received via a controller associated with theuser device, and wherein the interaction comprises a selection torotate, zoom in, zoom out, change color, change dimensions of thevirtual object, purchase the virtual object, view items related to thevirtual object, display the virtual object with one or more othervirtual objects, or a combination thereof.
 6. The method of claim 5,wherein the updated display comprises a rotated, zoomed in, zoomed out,color variation, or geometrically modified view of the virtual object.7. The method of claim 1, wherein the virtual object is displayed at theuser device and at the multimedia device simultaneously.
 8. The methodof claim 1, further comprising: prompting, by the user device, a displayof at least a portion of the augmented reality environment on themultimedia device, wherein the portion of the augmented realityenvironment includes background scenic features.
 9. The method of claim1, further comprising: prompting, by the user device, a display of apayment interface for purchase of the virtual object.
 10. The method ofclaim 9, further comprising: determining, by the user device,transaction specifications associated with the virtual object, thetransaction specifications including weight, price, deliveryinformation, color, geometric dimensions, promotional code(s), userreview(s), image(s) of the virtual object, size information, useinformation, material composition, nutritional information, dosage,serving size, ingredient information, warnings, warranty information, ora combination thereof; and prompting, by the user device, the display ofthe payment interface to include the transaction specifications.
 11. Auser device for prompting an updated display of a virtual object at amultimedia device, the user device comprising: one or more processors;and a data storage comprising instructions which, when executed by theone or more processors, cause the one or more processors to performoperations comprising: detecting the virtual object in an augmentedreality environment displayed; detecting the multimedia device withinthe augmented reality environment; prompting a display of the virtualobject at the multimedia device; receiving a user input comprising aninteraction with the virtual object; and prompting the updated displayof the virtual object at the multimedia device based on the user input.12. The user device of claim 11, wherein the user device comprises anaugmented reality device connected to the multimedia device via avirtual network.
 13. The user device of claim 11, wherein theinteraction is a user selection of the virtual object.
 14. The userdevice of claim 13, wherein the user selection comprises a touch gestureor a voice command.
 15. The user device of claim 11, wherein the userinput is received via a controller associated with the user device, andwherein the interaction comprises a selection to rotate, zoom in, zoomout, change color, change dimensions of the virtual object, purchase thevirtual object, view items related to the virtual object, display thevirtual object with one or more other virtual objects, or a combinationthereof.
 16. The user device of claim 15, wherein the updated displaycomprises a rotated, zoomed in, zoomed out, color variation, orgeometrically modified view of the virtual object.
 17. The user deviceof claim 11, wherein the virtual object is displayed at the user deviceand at the multimedia device simultaneously.
 18. The user device ofclaim 11, wherein the operations further comprise prompting the displayof at least a portion of the augmented reality environment on themultimedia device, and wherein the portion of the augmented realityenvironment includes background scenic features.
 19. The user device ofclaim 11, wherein the operations further comprise: prompting the displayof a payment interface for purchase of the virtual object; determiningtransaction specifications associated with the virtual object, thetransaction specifications including weight, price, deliveryinformation, color, geometric dimensions, promotional code(s), userreview(s), image(s) of the object, size information, use information,material composition, nutritional information, dosage, serving size,ingredient information, warnings, warranty information, or a combinationthereof; and prompting the display of the payment interface to includethe transaction specifications.
 20. A non-transitory computer readablemedium comprising instructions which, when executed by one or moreprocessors, cause the one or more processors to perform operationscomprising: detecting the virtual object in an augmented realityenvironment displayed; detecting the multimedia device within theaugmented reality environment; prompting a display of the virtual objectat the multimedia device; receiving a user input comprising aninteraction with the virtual object; and prompting the updated displayof the virtual object at the multimedia device based on the user input.